The present invention is directed to a method of verifying neural network mapping software, in general, and more particularly, to a method of verifying pretrained, static, feedforward neural network mapping software using Lipschitz constants for determining bounds on output values and estimation errors.
Certification of flight software containing neural network mapping functions generally encounters significant difficulties. Neural net mapping software has the reputation for probabilistic and unpredictable behavior. Such perception seems to be mainly a result of lack of understanding of the true nature of feedforward neural nets, in particular the distinction between the training process (which indeed can be unpredictable) and operation of a previously trained neural net (which is always deterministic).
What is needed is a finite testing procedure for verifying the behavior of pretrained neural net mapping software. Applicant has realized that the issue of how to train a neural net does not have to be addressed when analyzing the performance of the trained neural net software, rather it can be analyzed irrespective of the particular way it was trained. Applicant has discovered that pretrained neural net mapping software can be fully verified without any need to address the preceding training process. Static, feedforward neural nets represent purely deterministic mappings for which the output value can be fully predicted based on current instantaneous input values. While use of dynamic (recurrent) neural networks might understandably cause concerns about stability issues, static neural nets do not pose any such problems. Static neural network mapping software is merely a method of implementing multidimensional functions, just as look-up table mapping software is. Consequently, certification of a previously trained neural net should not be any more difficult than certification of any other mapping software, such as a look-up table. In this document, Applicant will focus on this analogy, and demonstrate mathematically that if a look-up table software implementation of a mapping is certifiable, so should be a neural net software implementation.
Therefore, Applicant""s focus here is to provide a verification method or testing procedure to guarantee proper behavior of static, feedforward neural network mapping software, especially for use under flight conditions.
In accordance with the present invention, a method of verifying pretrained, static, feedforward neural network mapping software having a neural net mapping function ƒ(x) that is intended to replace look up table mapping software having a look up table mapping function xcfx86(x) comprises the steps of: (1) establishing a multi-axis rectangular domain including upper and lower limits of each axis to bound all input vectors x of both said look up table mapping function xcfx86(x) and said neural net mapping function ƒ(x); (2)
determining a set of test points within the rectangular domain based on said upper and lower limits of each axis; (3) determining Lipschitz constant Kƒ for said neural net mapping function ƒ(x); and (4) determining upper and lower value bounds of said neural net mapping function ƒ(x) for said domain based on a function of said set of test points and said Lipschitz constant Kƒ, whereby if said upper and lower value bounds can be determined, then the neural network mapping software is verified.
In another aspect of the present invention, the foregoing described method includes the additional steps of: determining Lipschitz constant Kƒxe2x80x2 for the gradient ƒ(x)xe2x80x2; and determining upper and lower error bounds for the neural net and look up table functions based on a function of the set of test points and said Lipschitz constant Kƒxe2x80x2, whereby if the upper and lower value bounds and said upper and lower error bounds can be determined, then the neural network mapping software is verified.
In yet another aspect of the present invention, a method of verifying pretrained, static, feedforward neural network mapping software having a neural network mapping function ƒ("psgr"(x)+xcex6), where "psgr"(x) is a predetermined mapping of n dimensional state vectors x of a process into m dimensional measurement vectors z="psgr"(x), xcex6 is representative of a noise component of the process, and ƒ("psgr"(x)+xcex6) is an estimate of the desired output parameter y(est) of the process, comprises the steps of: (1) providing model mapping software including a predetermined mapping function xcfx86(x) for mapping said n dimensional state vectors x directly into a true value y(true) of said desired output parameter; (2) establishing a multi-axis rectangular domain including upper and lower limits of each axis to bound all state vectors x of both said predetermined mapping functions xcfx86(x) and "psgr"(x); (3) determining a set of test points within the rectangular domain based on said upper and lower limits of each axis; (4) determining Lipschitz constant Kƒ for neural net mapping function ƒ("psgr"(x)); (5) determining an upper bound xcex6up and a lower bound xcex6lo of the noise component xcex6; and (6) determining upper and lower value bounds of said neural net mapping function ƒ("psgr"(x)+xcex6) for said domain based on a function of said set of test points, said Lipschitz constant Kƒ and said upper bound xcex6up and lower bound xcex6lo, whereby if said upper and lower value bounds can be determined, then the neural network mapping software is verified.
In still another aspect of the present invention, the foregoing described method includes the additional steps of: (7) determining Lipschitz constant Kƒxe2x80x2 for the gradient ƒ("psgr"(x))xe2x80x2; and determining upper and lower error bounds for said neural net function ƒ("psgr"(x)+xcex6) and predetermined mapping function xcfx86(x) based on a function of said set of test points, said Lipschitz constant Kƒxe2x80x2, and said upper bound xcex6up and a lower bound xcex6lo, whereby if the upper and lower value bounds and said upper and lower error bounds can be determined, then the neural network mapping software is verified.